\begin{methods} 
\section{Methods} 
\subsection{General approach}  %%%% 'DONE', review done 
The basic idea, outlined in Figure~\ref{fig:pipeline}, is a separation of the discrimination between soluble and transmembrane proteins on the one hand and on the other hand the targeted prediction of transmembrane helices.
For simplification the level at which transmembrane proteins are to be separated from the soluble will further on be referred to as \textbf{SolTMH-prediction} and the helix assignment \textbf{Helix-prediction}.
Each of these two classification problems uses a distinct set of three SVMs that are joined in a jury decision. Features for these models mainly stem from the output of PredictProtein \citep{Rost2004a} and differ between the two classifiers
to match their specific demands.

As there is a possibility of misclassification the transmembrane helices are assigned independently of the soluble-transmembrane discrimination. Thus the user can decide for every prediction individually whether the helix prediction 
should be considered or not.
\begin{figure*}
 \includegraphics[scale=0.65]{./figures/pipeline.pdf}
 \caption{\textbf{TMH prediction pipeline.} Features are mainly extracted from PredictProtein. Afterwards a protein is classified as either soluble or transmembrane by the SolTMH-predictor. Additionally transmembrane helices are assigned by 
 the Helix-predictor. Both predictors employ a jury decision of three SVMs.
}
 \label{fig:pipeline}
\end{figure*}
\subsection{Dataset} %%%% 'DONE', review al:done maybe too long and extensive as we did not prepare the sets ..dontknow
The datasets for the SVMs training were chosen from UniProtKB \citep{Consortium2011}, selecting all sequences which were tagged with keyword kw-1133 as \textit{alpha-helical transmembrane} and not with keyword kw-1134 as \textit{transmembrane beta strand}.
In addition a cross-reference to a PDB structure had to be present, with the structure's sequence covering at least 85\% of the UniProt entry's sequence. The structure must also have been determined experimentally using X-Ray crystallography, NMR or EM.
Annotation of transmembrane helical residues is obtained from PDBTM \citep{Tusnady2005}. Annotation in this database is built on the structures ATOM record and therefore had to be mapped to the UniProt entry's sequence using a simple approach of 
mapping short segments of each sequence, inferring the rest and manually checking the resulting alignment. During this process residues in the UniProt sequence might not be assigned a structural class by PDBTM, since it is not aligned. In this case the residue
was considered to be outside of a transmembrane helix. In cases where information by PDBTM is available transmembrane helices and re-entrant loops are considered part of the transmembrane helix class and all other annotations are not.

For soluble sequences entries that die not match any of the above mentioned keywords were considered and further filtered using two regular expression for exclusion of membrane-associated proteins. The two resulting sets were redundancy reduced using 
UniqueProt at an HVAL cutoff of 0 \citep{Mika2003a}. This results in two final sets with 86 IMPs and 590 soluble proteins.

\subsection{Feature selection}  %%%% 'DONE', review al:done
As the time and computing resources were limited it was not possible to conduct a proper feature selection and thus the features were chosen on the basis of subject oriented literature.
The most important features from the PredictProtein output are the PSSM values. Here a window-size of 33 was chosen for the Helix-predictor, which is in accordance with previous results by \citealp{Nugent2009b}.
Further features from PredictProtein are overall secondary structure and solvent accessibility and with a small window, chemical properties like hydrophobicity, PSIC, DNA interaction (DISIS), protein 
interaction (ISIS), residue flexibility (PROFbval), secondary structure (PROFPhd), meta disorder, pfam domains and prosite patterns.

The SolTMH-predictor has as a much smaller feature set, with a window size of 21 for the PSSM values. In addition overall secondary structure and solvent accessibility, as well as DISIS, with a smaller window of size 5 complement the feature set.
Smaller window sizes did not improve performance and larger window sizes were not used in an effort to limit runtime of this predictor. The main aim of this predictor is not to deliver a good assignments of transmembrane helices
but predict tmh good enough to discriminate IMPs from soluble proteins.

\subsubsection{Additional features}  %%%% 'DONE', review al:done

Two additional features were used. The first one is the relative position of the helix in the whole sequence. 
Another additional feature is based on previous work by \citealp{Hessa2007a}. This feature was only employed for the SolTMH-prediction. \\ In the publication, a formula was developed for the prediction
of the apparent free energy of membrane insertion for a given sequence stretch.
The energy for a window of 19 residues is thereafter given as
\begin{equation}
\begin{split}
&\Delta G^{pred}_{app} = \sum^{19}_{i=1}{\Delta G^{aa(i)}_{app}} + \\
&0.27\sqrt{
\left(\sum^{19}_{i=1}{\Delta G^{aa(i)}_{app} \sin{100\degree i}}\right)^2 +
\left(\sum^{19}_{i=1}{\Delta G^{aa(i)}_{app} \cos{100\degree i}}\right)^2}
\end{split}
\end{equation}
The additional parameters $c_1, c_2, c_3$ given in the original publication were ignored since they deteriorated the results.

\subsection{Model building and cross validation} 
Models for the SolTMH and Helix-predictor were build separately using distinct training sets as outlined in Figure~\ref{fig:trainingsets}. The training set for the Helix-predictor does not contain any soluble proteins that might have a detrimental effect
on the prediction of transmembrane regions. However this also leads to a necessity for a second predictor that handles discrimination to soluble proteins. Used here is an approach similar to the Helix-predictor, using a jury decision of three SVMs. An
additional method was tested using a fixed window size approach based on the work by \citeauthor{Hessa2007a} mentioned before, in combination with an energy cutoff. This predictor yielded comparable results at a much shorter runtime, however the 
optimisation of only the cutoff is highly prone to overfitting and was considered too unstable for future usage.

\begin{figure}
 \includegraphics[scale=0.28]{./figures/sets.pdf}
 \caption{\textbf{SVM training sets.} Shown is the training-set preparation for the distinct SVMs.}
 \label{fig:trainingsets}
\end{figure}

The models were trained in a three-fold cross-validation, assigning subsets as outlined in \ref{tbl:setmatching}. This ensures that training, parameter optimisation and testing are done on three independent sets, aiming to make maximum use of the limited
amount of data available. Oversampling and application of the SMOTE \citep{Chawla2002} was used to counteract the uneven distribution of classes in the training data but did not improve results.

\begin{table}[!t]
\processtable{\textbf{Allocation of sets in 3-fold cross-validation}. Shown is the rotation of the three independent sets during cross validation. If the SVM trained on the first set, then the parameters were optimized 
in the grid search using the second set and finally performance was assessed using the third set. This assignment holds true for both the SolTMH- as well as the Helix-predictor\label{tbl:setmatching}}
{\begin{tabular}{p{2.2cm}p{2.2cm}p{2.2cm}}\toprule
\textbf{Training} & \textbf{Optimisation} & \textbf{Test}\\
\midrule
1 & 2 & 3\\
2 & 3 & 1\\
3 & 1 & 2\\
\botrule
\end{tabular}}{}
\end{table}

\subsection{Scoring} %%%% 'DONE', review al:done
The discrimination between soluble proteins and IMPs is a binary decision. Based on these results Sensitivity, Specificity and MCC were calculated  as follows
\begin{equation}
\text{Sens} = \frac{\text{TP}}{\text{TP} + \text{FN}}
\end{equation}
\begin{equation}
\text{Spec} = \frac{\text{TN}}{\text{TN} + \text{FP}}
\end{equation}  
\begin{equation}
\text{MCC} = \frac{\text{TP} \cdot \text{TN}-\text{FP} \cdot \text{FN}}{\sqrt{(\text{TP}+\text{FN})(\text{TP}+\text{FP})(\text{TN}+\text{FP})(\text{TN}+\text{FN})}}
\end{equation} 

For the assessment of transmembrane helix prediction several of the de-facto standard scores were employed as described by \citeauthor{Chen2002}. The scores, shown in Table~\ref{tbl:level2scores}, can be divided into two groups:
Per-residue scores evaluate performance on the basis of single residues, not taking into account the surrounding area. Per-segment scores measure performance on a higher level.
Here complete helices are compared between prediction and observation. This scoring is more sound in that it takes into account the problems of transmembrane segment annotation. Even given a high resolution structure of a protein it is not
trivial, and to some extent a matter of definition, to determine at which exact residue a helix begins or ends. Therefore two helices, one observed and one predicted, count as correctly predicted if at least three residues overlap. However as established by
\citeauthor{Chen2002} if multiple predicted helices have sufficient overlap only one counts as correctly predicted and if a predicted helix sufficiently overlaps with two observed helices it only counts once. The resulting scores are harder to master
since they are not dominated by non-transmembrane residues, as are many of the per-residue scores, and severely punish single short or unconnected helices.

\begin{table}[!t]
\processtable{\textbf{Scores employed for the performance assessment of the Helix-predictor.}\label{tbl:level2scores}}
{
  \begin{tabular}{ll}
\toprule
\textbf{Score} & \textbf{Formula}\\
\midrule
\textbf{Q\textsubscript{ok}} & $\frac{1}{N_{prot}}\sum^{i=1}_{N_{prot}}{\delta_{i}} ; \delta_{i}  = \begin{cases}1 &\text{,if } \textbf{Q}_{\textbf{htm}}^{\textbf{\%obs}} = 1 =  \textbf{Q}_{\textbf{htm}}^{\textbf{\%pred}}\\0 & \text{, otherwise}\end{cases}$\\[3ex]
$\textbf{Q}_{\textbf{htm}}^{\textbf{\%obs}}$ & $\frac{\text{number of correctly predicted TMHs in data set}}{\text{number of TMHs observed in data set}}$\\[3ex]
$\textbf{Q}_{\textbf{htm}}^{\textbf{\%pred}}$ & $\frac{\text{number of correctly predicted TMHs in data set}}{\text{number of TMHs predicted in data set}}$\\[3ex]
\textbf{Q\textsubscript{2}} & $\frac{1}{N_{prot}} \cdot \sum_{i=1}^{N_{prot}}{\frac{\text{number of residues predicted correctly in protein i}}{\text{number of residues in protein i}}}$\\[3ex]
$\textbf{Q}_{\textbf{2T}}^{\textbf{\%obs}}$ & $\frac{\text{number of residues correctly predicted in TMHs}}{\text{number of residues observed in TMHs}}$\\[3ex]
$\textbf{Q}_{\textbf{2T}}^{\textbf{\%pred}}$& $ \frac{\text{number of residues correctly predicted in TMHs}}{\text{number of residues predicted in TMHs}}$\\ [3ex]
$\textbf{Q}_{\textbf{2N}}^{\textbf{\%obs}}$& $\frac{\text{number of residues correctly predicted in non-TMHs}}{\text{number of residues observed in non-TMHs}}$\\ [3ex]
$\textbf{Q}_{\textbf{2N}}^{\textbf{\%pred}}$& $\frac{\text{number of residues correctly predicted in non-TMHs}}{\text{number of residues predicted in non-TMHs}}$\\
\botrule
\end{tabular}}{\textbf{Q\textsubscript{ok}}, $\textbf{Q}_{\textbf{htm}}^{\textbf{\%obs}}$ and $\textbf{Q}_{\textbf{htm}}^{\textbf{\%pred}}$ are per-segment scores. All others are per-residue scores.}
\end{table}

\subsection{Parameter Optimisation}
All SVMs use the standard RBF kernel. Other kernels were not evaluated due to time constraints.
The cost and gamma parameters of each SVM were optimised using a grid search approach with an exponential step size as suggested by a LIBSVM tutorial \citep{Hsu2010}. For the discrimination a further parameter was adjusted, 
namely the criterion by which it is decided whether the originally per-residue classification of the SolTMH-predictor 
denotes a soluble protein or IMP.
The sum over the SVM confidences of predicted TMH residues within one protein, was chosen as appropriate parameter, and subsequently optimised together with C and gamma.
For this purpose, the Mathews Correlation Coefficient (MCC) was employed as measure for the determination of the optimal parameter combination. Sensitivity and Specificity were regarded as well. 

The Helix-predictor, which aims at completely different aspects of the problem, was optimised towards Q\textsubscript{ok} while also taking into account Q\textsubscript{2}, $\text{Q}_{\text{htm}}^{\text{\%obs}}$ and 
$\text{Q}_{\text{htm}}^{\text{\%pred}}$.
The Q\textsubscript{ok} changes within the grid search are displayed in Figure~\ref{fig:heat}. Some parameter combinations yield very unfavourable results (red) while parameter combinations around $C=-1$ and $\gamma=-7$ (green) show a good performance.
The results of the whole optimisation are shown in Table~\ref{tbl:paramopt}
\begin{figure*}
\centering
 \includegraphics[scale=0.13]{./figures/GridHeat.png}
 \caption{\textbf{Results of second level parameter optimization.} Shown are the resulting Q\textsubscript{ok} scores for the grid search performed on each of the three SVMs for the prediction of transmembrane helices.
 C and gamma are the cost and gamma parameters for the SVM and are given as $\log_2$.
 For each training set, the optimisation set is chosen as denoted in Table~\ref{tbl:setmatching}.
}
 \label{fig:heat}
\end{figure*}

\begin{table}[!t]
\processtable{\textbf{Optimisation results.} Optimized parameters for the SolTMH- and Helix-predictors. \textbf{C} and \textbf{Gamma} are given as $\log_2$.\label{tbl:paramopt}}
{\begin{tabular}{p{2.6cm}p{1.5cm}p{1.5cm}p{1.5cm}}\toprule
  &  \textbf{C} & \textbf{Gamma} & \textbf{Cutoff} \\
\midrule
SolTmh-predictor1 & 1 & -7 & 4 \\
SolTmh-predictor2 & 1 & -11 & 4\\
SolTmh-predictor3 & 1 & -7 & 2\\
Helix-predictor1 &  -2 & -7 & -\\
Helix-predictor2 &  -1 &  -7 & -\\
Helix-predictor3 &  -1 &  -7 & -\\
\botrule
\end{tabular}}{}
\end{table}

\subsection{Jury decision and output format} 
Every SVM of the SolTMH-predictor has its own cutoff that is applied to the sum over all internal SVM confidences for a predicted TMH residues $r$. This is normalized to a TMH-probability between 0 and 1 as follows
\begin{equation}
  \frac{\sum_{i=1,\atop r \in \text{TMH} }^{N} \text{conf}(r_i)}{2\cdot\text{cutoff}}     % TODO define N in text afterwards, why the strange formatting in the sum index. Generell check cih die formel nich wirklich, zu spaet
\end{equation}
It is important to note here, that if the confidence sum in the numerator equals $2 \cdot \text{cutoff}$, it is not raised any further to ensure a equal distribution and normalisation in dependence of the cutoff. It also ensures that no single 
SVM in the jury can dominate the other two, as finally, the average over the three normalized TMH-probabilities determines the final classification. A protein with a value greater than $0.5$ is considered an IMP.

For the jury on helix level simply an average over the SVM-confidence is built and according to it a residue is assigned to a transmembrane helix if the value is greater than 0.5.

The output combines the SolTMH-prediction and Helix-prediction, in that at first the classification of the protein class is given and then separately below the helix assignment, as outlined in Figure~\ref{fig:pipeline}.
\end{methods}

